Loss Bounds for Uncertain Transition Probabilities in Markov Decision Processes

We analyze losses resulting from uncertain transition probabilities in Markov decision processes with bounded nonnegative rewards. We assume that policies are pre-computed using exact dynamic programming with the estimated transition probabilities, but the system evolves according to different, true transition probabilities. Our approach analyzes the growth of errors incurred by stepping backwards in time while precomputing value functions, which requires bounding a multilinear program. We present loss bounds for the finite horizon undiscounted, finite horizon discounted, and infinite horizon discounted cases. Loss bounds are given in terms of the maximum total variation error of all transition probabilities, maximum reward, number of stages, and discount factor. A tight example is given for the finite horizon undiscounted case.